Proximal Mapping for Symmetric Penalty and Sparsity

This paper studies a class of problems consisting of minimizing a continuously differentiable function penalizedwith the so-called `0-norm over a symmetric set. These problems are hard to solve, yet prominent in many fields and applications.We first study the proximal mapping with respect to the `0-norm over symmetric sets, and provide an efficient method to attainit. The method is then improved for symmetric sets satisfying a sub-modularity-like property, which we call “second ordermonotonicity” (SOM). It is shown that many important symmetric sets, such as the `1, `2, `∞-balls, the simplex and the full-simplex, satisfy this SOM property. We then develop, under the validity of the SOM property, necessary optimality conditions,and corresponding algorithms that are guaranteed to converge to points satisfying the aforementioned optimality conditions. Weprove the existence of a hierarchy between the optimality conditions, and consequently between the corresponding algorithms.